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This listing of claims will replace all prior versions, and listings, of claims in the application: 
Listing of Claims: 

1 . (currently amended) An automated system for experimentation comprising: 
a communication management svstem n oxporimont engine operable to dofine [ an 

experiment re l ating toj various trcatmonto for o sot of content elements, tho experiment ongino 
opera b l e t o pgnrfnrfr fhf» r>vpr>rimnnt nvnrn data notwor k supporti n q the creation and execution 
of one or more experiments to test the behavior of users to various treatments for a set of 
content elements: and 

an allocator module operable over a data network to systematically allocate a treatment 
to a user in connection with an experiment. 

an observation modulo operab le tO ' Oolloct observation data relating to user behavior for 
oach treatment; and 

a scripting / cohcduling ongine opomb l o to coord i nate tho operation of the experiment 

engine and the observation modu l o . 

2. (currently amended) The system of Claim 1 wherein the communica tion management 
system cxperimont engine is operable to identify elements of content which may influence user 
behavior. 

3. (currently amended) The .system of Claim 1 wherein the communication management 
svstem e xpcrimcnf ong i ne is operable to implement statistical sampling procedures to deliver 
over the data network the various treatments to respective control groups of users. 
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4. (currently amended) The system of Claim 1 wherein the communication management 
system oxporimont eng i ne is operable to store data relating to the set of content elements, 
objective user behavior to be observed, and users allocated to receive treatment during the 
experiment. 

5. (currently amended) The system of Claim 1 wherein the communication management 
system o xporim e nt e ng i ne comprises an experiment manager object operable to control the 
execution of the experiment. 

6. (currently amended) The system of Claim 1 wherein the communica tion management 
system o xpor i mont engine comprises an experiment manager object operable to specify 
different treatments of content 

7. (currently amended) The system of Claim 1 wherein the communication mana gement 
system ex periment ong i no comprises an experiment manager object operable to define and 
implement statistical sampling procedures. 

8. (currently amended) The system of Claim 1 wherein the communication management 
system oxporimont ongino comprises an experiment manager object operable to record the 
experiment under way and the (participating users. 

9. (currently amended) The system of Claim 1 wherein the communication management 
system oxporiment e ngin e comprises an experiment data store operable to store experiment 
data. 
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10. (original) The system of Claim 1 wherein the data network comprises the Internet 

1 1 . (currently amended) The system of Claim 1 where i n th efurther co m prising an 
observation module operable to collect observation data relating to actual b ehavior of users to 
each treatment e emprisoG an observation access objeQt operable to provide access to the 
oboorvation data . 

12. (currently amended) The system of Claim 1 wherein the communica tion management 
system oxporimont engine is operable to generate a set of experiment rules for allocating 
treatments during the experiment. 

13. (currently amended) The system of Claim 1 further comprising an interface in 
communication with the communication management system oxporimont eng i ne , the interface 
operable to allow a user to interact with the communication management system experiment 
engine to define the experiment. 

14. (cancelled) 

1 5. (currently amended) An automated method experimentat i on comprising: 

defining an experimen t to gauge user reaction- re l ating to various treatments for a set of 
content elements; 

conducting the experiment over thea data network;_and 

collecting over the data network observation data relating to user behavior for each 
treatment^and 
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g e ne rating at loaot ono ooript to coordinate defining an o xperimont. conducting tho 

cxporiroont, and co l lecting obocrvation data . 

16. (original) The method of Claim 15 wherein defining an experiment comprises: 
identifying desired objectives for user behavior; 

identifying which treatments may influence user behavior related to the desired 
objectives; and 

generating the various treatments using different combinations of the content elements. 



17. (original) The method of Claim 16 further comprising: 

defining at least one control variable for the various treatments; and 
assigning a respective level for the control variable for each treatment. 

1 8. (original) The method of Claim 15 wherein conducting the experiment comprises: 

selecting at least one treatment for delivery to users; 

grouping users into a plurality of segments, each segment comprising users with similar 
behavioral characteristics; and 

specifying a particular segment of users to receive the selected treatment. 

1 9. (original) The method of Claim 1 5 wherein conducting the experiment comprises: 

statistically sampling to specify a plurality of control groups, each control group 
comprising at least one user; 

specifying a particular treatment to be delivered to the at least one user in each control 




group; 
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receiving identical requests for content from the respective at least one user in each of 
the control groups; and 

in response to the identical requests, delivering to the at least one user in each control 
group a different treatment 

20. (original) The method of Claim 1 9 further comprising observing site-related behavior of 
users receiving the. various treatments. 

21. (original)" The method of Claim 15 wherein the data network comprises the Internet 

22. (original) The method of Claim 20 wherein collecting observation d£ta comprises recording 
observed behavior in each control group during the experiment. 

23. (original) The method of Claim 15 further comprising identifying elements of content which 
potentially influence behavior of users. 

24. (currently amended) An automated system for oxporimontatioR -comprising: 

a content system operable to store content, the content comprising a set of content 
elements; and 

a communication management system in communication with the content system and 
operable to define an experiment relating to various treatments for the set of content elements, 
conduct the experiment over a data network, collect over the data network observation data 
relating to user behavior for each treatment and generate at least one script for coordinating the 
operation of the content system and the communication management system. 
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25. (original) The system of Claim 24 wherein the communication management system is 
operable to identify content elements which may influence user behavior. 

26. (original) The system of Claim 24 wherein the communication management system is 
operable to implement statistical sampling procedures to deliver over the data network the 
various treatments to respective control groups of users. 

27. (original) The system of Claim 24 wherein the communication management system is 
operable to generate a set of experiment rules for allocating treatments during the experiment 

28. (original) The system of Claim 24 wherein the content system is operable to allocate 
treatments to users according to the experiment rules. 

29. (original) The system of Claim 24 wherein the communication management system is 
operable to group users into a plurality of segments, each segment comprising users with . 
similar behavioral characteristics. 

30. (original) The system of Claim 24 wherein said data network comprises the Internet. 

31. (original) The system of Claim 24 wherein the communication management system is 
operable to define at least one control variable for the various treatments and to assign a 
respective level for the control variables for each treatment 

32. (original) The system of Claim 24 wherein the content system and the communication 
management system are connected to the data network. 
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33. (original) The system of Claim 24 wherein the content system comprises an allocator 
module operable to support an interface with the communication management system. 

34. (original) The system of Claim 24 wherein the communication management system 
comprises an allocator Interface object operable to support an interface with the content system. 

35. (original) The system of Claim 24 further comprising a content provider interface operable 
to support an interface between the communication management system and a manager user. 

36. (original) An automated system for experimentation comprising: 

an experiment engine operable to define an experiment relating to various treatments for 
a set of content elements, the experiment engine operable to allocate each treatment to a 
separate control group of users over a data network; 

an observation module operable to collect observation data relating to user behavior for 
each treatment; and 

a scripting / scheduling engine operable to coordinate the operation of the experiment 
engine and the observation module. 

37. (original) The system of Claim 36 wherein the experiment engine is operable to identify 
elements of content which may influence user behavior. 

38. (original) The system of claim 36 wherein the experiment engine is operable to implement 
statistical sampling procedures to deliver over the data network the various treatments to 
respective control groups of users. 
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39. (original) The system of Claim 36 wherein the data network comprises the Internet 

40. (original) An automated method for experimentation comprising: 

defining an experiment relating to various treatments for a set of content elements; 
allocating each treatment to a separate control group of users over a data network; 
collecting over the data network observation data relating to user behavior for each 
treatment; and 

generating at least one script to coordinate defining an experiment, conducting the 
experiment and collecting observation data. 

41 . (original) The method of Claim 40 wherein the data network comprises the Internet 

42. (original) The method of Claim .40 further comprising Identifying elements of content which 
potentially influence behavior of users. 

43. (original) A method for on-line experimentation comprising: 

defining a plurality of treatments for a set of content elements; 
providing a website on a data network; 

receiving requests for content from users accessing the website on the data network; 
statistically sampling to create at least one control group of users; 
allocating over the data network a first treatment to each user in the control group; 
allocating over the data network a second treatment to each user not in the control 

group; 
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collecting observation data for observed behavior of users in the control group and users 
not in the control group; and 

generating at least one script to coordinate allocating the first and second treatments 
and collecting observation data. 

44. (original) The method of Claim 43 further comprising identifying elements of content which 
may influence user behavior. 

45. (original) The method of Claim 43 wherein providing a website comprises providing a web 
page at which the set of content elements is available in the form of any of the treatments. 

46. (original) The method of Claim 43 wherein the data network comprises the Internet. 

47. (new) The system of claim 1 wherein the set of content elements are content elements 
for a web page. 

48. (new) The system of claim 47 wherein the data network is the Internet. 

49. (new) The system of claim 2 wherein content comprising a plurality of the content 
elements is configured to influence user behavior in accordance with observations from an 
experiment. 

50. (new) The system of daim 1 wherein the system Includes a mode! engine to determine 
what set of content elements is most suitable for achieving a desired outcome. 

Page 1 1 - RESPONSE TO OFFICE ACTION DATED JULY 1 , 2003 
Serial No. 09/648,429 




PAGE 17/27 ' RCVD AT 1/2(2004 4:48:18 PM [Eastern Standard Time] ■ SVR:USPTMFXRF.2/1 ■ DNIS:7467239 8 CSD:5032962172* DURATION (mm«ss):08-30 



* JAN/02/2004/FRI 02:53 PM GANZ LAW FAX No. 5032962172 P. 018/027 



51 . (new) An online experimentation system configured to create various treatments for a 
set of content elements and to allocate the treatments to users to test user behavior in response 
to the various treatments, the system being further configured to capture data relating to the 
observed behavior of users relative to the various treatments. 

52. (new) The system of claim 51 wherein the system is configured to assess how content 
element variables drive user choices or behavior. 

53. (new) The system of claim 51 wherein the system is configured to generate a set of 
rules that dictate how treatments are allocated to particular users. 

54. (new) A computer implemented method comprising: 

creating various treatments for a set of content elements and to allocate the treatments 
to users to test user behavior in response to the various treatment; 
allocating the treatments to users; and 

capturing data relating to the observed behavior of users relative to the various 
treatments. 

55. (new) The method of claim 54 further comprising assessing from captured data how 
content element variables drive user choices or behavior. 

56. (new) The method of claim 54 further comprising generating a set of rules that dictate 
how treatments are allocated to particular users. 

57. (new) An machine readable medium including a set of instructions for: 
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creating various treatments for a set of content elements and to allocate the treatments 
to users to test user behavior in response to the various treatment; 
allocating the treatments to users; and 

capturing data relating to the observed behavior of users relative to the various 
treatments. 



58. (new) The machine readable medium of claim 57 wherein the medium includes 
instructions for assessing from captured data how content element variables drive user choices 
or behavior. 

59. (new) The machine readable medium of claim 57 wherein the medium further includes 
instructions for generating a set of rules that dictate how treatments are allocated to particular 
users. 

60. (new) The system of claim 1 wherein a treatment is allocated to a user according to 
experiment rules, prediction rules, or user defined rules. 

61 . (new) The method of claim 15 wherein treatments are allocated to users according to 
experiment rules, prediction rules, or user defined rules. 

62. (new) The medium of claim 57 wherein the allocating is according to experiment rules, 
prediction rules, or user defined rules. 

63. (new) The computer readable medium of claim 58 wherein the instructions are 
included for analyzing captured data using a multivariate statistical modeling technique to 
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determine what content elements in a treatment influenced the probability of an outcome in 
terms of users behavior or reaction to treatments. 

64. (new) The system of claim 1 wherein the system is operable to create and execute a 
designed experiment s 

65. (new) The system of claim 1 1 wherein a model engine is in communication with the 
observation module, the model engine operable to generate a model for predicting anticipated 
behavior of users based upon the observation data relating to actual behavior. 

66. (new) The system of claim 65 further comprising a model engine in communication with 
the observation module, the model engine operable to generate a model for predicting 
anticipated behavior of users based upon the observation data relating to actual behavior. . 

67. (new) The system of claim 66 further comprising, a prediction engine in communication 
with the model engine, the prediction engine operable to deliver one of the treatments to users 
according to the model in order to optimize a desired outcome. 

68. (new) The system of Claim 67 wherein the prediction engine implements a personalization 
process. 

69. (new) The system of Claim 68 wherein the prediction engine segments users into a 
plurality, of segments, each segment comprising users with similar behavioral characteristics. 
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70. (new) The system of Claim 69 wherein the prediction engine is operable to generate a 
set of prediction rules for allocating treatments to users with specific behavioral characteristics. 

71 . (new) The system of Claim 70 comprising an interface in communication with the 
experiment engine, the interface operable to allow a user to interact with the experiment engine 
to define the experiment. 

72. (new) A method performed on a computer for automatically managing communication 
comprising: 

defining an experiment relating to various treatments for a set of content elements; 
conducting the experiment over a data network; 

collecting over the data network observation data relating to actual behavior of users to 
each treatment; and 

generating a model for predicting anticipated behavior of users based upon the 
observation data relating to actual behavior; and 

generating at least one script to coordinate defining an experiment, conducting the 
experiment, collecting data, and generating a model. 

73. (new) The system of Claim 24 wherein the communication management system is 
operable to generate a set of prediction rules for allocating treatments to users with specific 
behavioral characteristics. 

74. (new) The system of Claim 73 wherein the content system is operable to allocate 
treatments to users according to the prediction rules. 
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75. (new) The system of Claim 24 wherein the communication management system 
implements a personalization process. 

76. (new) The method of claim 15 wherein the experiment is delivered over the data 
network to respective control groups of users according to the experiment and wherein the 
method further comprises generating a model for predicting anticipated behavior of users based 
upon the observed behavior; and generating a prediction for delivering one of the treatments to 
users according to the model in order to optimize a desired outcome. 

77. (new) The method of Claim 76 wherein defining an experiment comprises identifying 
elements of content which potentially influence behavior of users. 

78. (new) An automated system for delivery of content, comprising: 

a communication management system including a prediction engine configured to 
generate predictions on how users who are to be presented a treatment 
comprising a set of content elements will react to the treatment the prediction 
engine generating predictions based on data collected from one or more 
experiments that tested user behavior to content elements in the set, the 
communication management system delivering treatments to users in 
accordance with the predictions to increase the probability of a desired user 
behavior in response to the presented treatment. 

79. (new) An automated method, comprising: 

generating predictions on how users who are to be presented a treatment comprising a 
set of content elements will react to the treatment, the predictions being 
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generated from data collected from one or more experiments that tested user 
behavior to content elements in the set; and 
delivering treatments to users in accordance with the predictions to increase the 

probability of a desired user behavior in response to the presented conduct. 




80. (new) A machine readable medium including a set of instructions for: 

generating predictions on how users who are to be presented a treatment 
comprising a set of content elements will react to the treatment, the 
predictions being generated from data collected from one or more 
experiments that tested user behavior to content elements in the set; and 

delivering treatments to users in accordance with the predictions to increase the 
probability of a desired user behavior in response to the presented 
conduct 

81 . (new) A content page comprising a set of content elements generated by a prediction 

engine, the prediction engine generating predictions based on data collected from one or 
more experiments that tested user behavior in response to content elements in the set 

82. (new) The system of claim 64 wherein the designed experiment reduces the number of 

combinations included in the experiment by measuring and estimating only those effects 
that are of a predetermined interest. 
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